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DETAILED ACTION 

1 . This action is responsive to communications: amendment filed 4/25/2005. 

2. Claims 1-28 are pending in the case. Claims 1,16, and 22 are independent claims. 

Claim Objections 

3. Claims 5 and 20 are objected to under 37 CFR 1.75(c), as being of improper dependent 
form for failing to further limit the subject matter of a previous claim. Applicant is required to 
cancel the claims, or amend the claims to place the claims in proper dependent form, or rewrite 
the claims in independent form. The maximal disambiguation optimization criterion limitation 
of claims 5 and 20 is a redundant limitation to independent claims 1 and 16. 

Claim Rejections - 35 USC §103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
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invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 



5. Claims 1-28 are rejected under 35 U.S.C 103(a) as being unpatentable over Stubler 
et al. (hereinafter "Stubler"), US 6,804,684 B2 filed 5/7/2001 in view of Toyama, US 
6,816,847 Bl filed 9/23/1999. 

Regarding independent claim 1, 16, and 22, Stubler teaches actively selecting 
examples of multimedia content to be annotated by a user and accepting input annotations from 
the user for the selected examples in fig. 2, fig. 6-7, col. 3 line 46 - col. 4 line 12, col. 8 lines 18- 
23, and col. 9 line 65 - col. 10 line 18. Stubler teaches propagating the input annotations to other 
instances of multimedia content and storing the input annotations and the propagated annotations 
in fig. 2, col. 3 line 46 - col. 4 line 12, and col 8 lines 18-55. 

Stubler does not teach actively selecting examples of multimedia content to be annotated 
by a user, wherein the examples of multimedia content are selected based on at least one 
criterion for achieving a maximal disambiguation result. Toyama does teach actively selecting 
examples of multimedia content to be annotated by a user, wherein the examples of multimedia 
content are selected based on at least one criterion for achieving a maximal disambiguation result 
in col. 5 lines 15-46. 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards propagating 
annotations from stored images to similar images that have not yet been annotated. Toyama 
teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards propagating annotations from a 
known training set to similar images that have do not have any associated annotations. Thus, the 
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field of endeavor is very similar for Stubler and Toyama. It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to have combined the teachings of 
Stubler and Toyama to have created the claimed invention. It would have been obvious and 
desirable to have used the active selection techniques disclosed by Toyama in col. 5 lines 15-46 
to have improved the active selection of images to be annotated as taught by Stubler so that a 
maximal disambiguation result could have been achieved as is the reason for a Toyama selection 
of a wide variety of images in col 5 lines 30-32 and a wide variety of annotations for the images 
in col. 5 lines 32-46. 

Regarding dependent claims 2 and 17, Stubler teaches wherein the step of actively 
selecting is performed using a selection technique from the group consisting of deterministic and 
probabilistic in col. 4 line 64 - col. 5 line 19. 

Regarding dependent claims 3 and 18, Stubler teaches wherein the step of actively 
selecting which is performed deterministically or probabilistically, is based on explicit models 
and feature proximity/similarity measures, and returns one or more examples of multimedia 
content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 4 and 19, Stubler teaches wherein the step of actively 
selecting, which is performed deterministically or probabilistically, is based on implicit models 
and feature proximity/similarity measures, and returns one or more examples of multimedia 
content to be annotated in fig. 2 and col. 8 lines 18-55. 

Regarding dependent claims 5 and 20, Stubler teaches wherein an optimization 
criterion for active selection includes one or more criteria selected form the group consisting of: 
information measures, and confidence in fig. 2 and col. 8 lines 18-55. 
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Regarding dependent claims 6 and 21, Stubler teaches wherein the multimedia content 
comprises one or more types selected from the group consisting of: images, audio, video, 
graphics, text, multimedia, Web pages, time series data, surveillance data, sensor data, relational 
data, and XML data in col 3 line 46 - col. 4 line 12. 

Regarding dependent claim 7, Stubler teaches wherein the input annotations are created 
by a user with reference to a vocabulary in col 3 line 46 - col. 4 line 12 and col. 8 lines 18-23. 

Regarding dependent claim 8, Stubler teaches wherein the vocabulary contains one or 
more items selected from the group consisting of: terms, concepts, labels, and annotations in col. 
3 line 46 - col. 4 line 12 and col. 8 lines 18-23. 

Regarding dependent claim 9, Stubler teaches wherein the process of creating input 
annotations by the user involves multimodal interaction with the user using graphical, textual, 
and/or speech interface in fig. 2 and col. 3 line 46 - col. 4 line 12. 

Regarding dependent claim 10, Stubler teaches wherein the input annotations are 
created by means of steps selected from the group consisting of: creating new annotations, 
deleting existing annotations, rejects proposed annotations, and modifying annotations in fig. 6-7 
and col. 9 line 65 - col. 10 line 18. 

Regarding dependent claim 11, Stubler teaches wherein the vocabulary is adaptively or 
dynamically organized and/or limited by the system of the user in fig. 6-7 and col. 9 line 65 - 
col 10 line 18. 

Regarding dependent claim 12, Stubler teaches wherein the multimodal interaction 
involves speech recognition, gaze detection, finger pointing, expression detection, and/or 
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effective computing methods for sensing a user's state in fig. 6-7 and col. 9 line 65 - col. 10 line 
18. 

Regarding dependent claim 13, Stubler teaches wherein the determination of the 
propagation of annotations is made deterministically or probabilistically and on the use of 
' models for each annotation or for joint annotations in col. 4 line 64 - col. 5 line 19. 

Regarding dependent claim 14, Stubler teaches wherein the models are created or 
learned automatically or semi-automatically and/or are updated adaptively from interaction with 
the user in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Regarding dependent claim 15, Stubler teaches wherein the models are based on 
nearest neighbor voting or variants, parametric or statistical models, expert systems, rule-based 
systems, or hybrid techniques in fig. 2, col. 3 line 46 - col. 4 line 12, and col. 8 lines 18-55. 

Regarding dependent claims 23, 25, and 27, Stubler does not teach wherein that at least 
one criterion includes an ambiguity level of the selected examples. Toyama does teach wherein 
that at least one criterion includes an ambiguity level of the selected examples in col. 5 lines 15 - 
col. 6 line 50. Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards 
propagating annotations from stored images to similar images that have not yet been annotated. 
Toyama teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards propagating 
annotations from a known training set to similar images that have do not have any associated 
annotations. Thus, the field of endeavor is very similar for Stubler and Toyama. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to have 
combined the teachings of Stubler and Toyama to have created the claimed invention. It would 
have been obvious and desirable to have used the active selection techniques disclosed by 
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Toyama in col. 5 lines 15-46 to have improved the active selection of images to be annotated as 
taught by Stubler so that a maximal disambiguation result could have been achieved as is the 
reason for a Toyama selection of a wide variety of images in col 5 lines 30-32 and a wide 
variety of annotations for the images in col. 5 lines 32-46. 

Regarding dependent claims 24, 26, and 28, Stubler does not teach wherein the at least 
one criterion includes a confidence level of the selected examples, the confidence level being 
inversely proportional to a distance of a new feature of the selected examples from a separating 
hyperplane in an induced higher dimensional feature space. Toyama discloses the use of support 
vector machines in col. 6 lines 10-50. Toyama discloses wherein the at least one criterion 
includes a confidence level of the selected examples, the confidence level being inversely 
proportional to a distance of a new feature of the selected examples from a separating hyperplane 
in an induced higher dimensional feature space in col. 5 lines 15 - col. 6 line 50. 

Stubler teaches in col. 2 line 59 - col. 3 line 10 that it is directed towards propagating 
annotations from stored images to similar images that have not yet been annotated. Toyama 
teaches in col. 1 line 47 - col. 2 line 14 that it is directed towards propagating annotations from a 
known training set to similar images that have do not have any associated annotations. Thus, the 
field of endeavor is very similar for Stubler and Toyama. It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to have combined the teachings of 
Stubler and Toyama to have created the claimed invention. It would have been obvious and 
desirable to have used the active selection techniques disclosed by Toyama in col. 5 lines 15-46 
to have improved the active selection of images to be annotated as taught by Stubler so that a 
maximal disambiguation result could have been achieved as is the reason for a Toyama selection 
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of a wide variety of images in col. 5 lines 30-32 and a wide variety of annotations for the images 
in col. 5 lines 32-46. 

Response to Argum ents 

6. Applicant's arguments with respect to claims 1-28 have been considered but are moot in 
view of the new ground(s) of rejection. Applicant's amendment to independent claim 1 and 16 
significantly changed the scope of the claimed invention when interpreted as a whole. The 
Examiner notes the similarity of the added limitation to that of dependent claims 5 and 20. 
However, a change in the scope occurs, as the claimed "maximal disambiguation" is now recited 
as a requirement within claims 1 and 16 as opposed to being offered as a choice within (Markush 
type) claims 5 and 20. The Examiner has searched and found the prior art reference of Toyama 
which teaches actively selecting examples of multimedia content, wherein the examples of 
multimedia content are selected based on at least one criterion for achieving a maximal 
disambiguation result. Toyama also teaches use of a support vector machine for determining 
ambiguity. The Examiner believes the combination of teachings of Stubler et al. and Toyama 
render the claimed invention obvious. 

Conclusion 

7. The prior art made of record and not relied upon is considered pertinent to Applicant's 
disclosure. K. Goh, E. Chang, and K. Cheng, "Support vector machine pairwise classifiers with 
error reduction for image classification", Proceedings of the 2001 ACM workshops on 
Multimedia, September 2001, pages 32-37 teaches applying support vector machines to image 
classification. Piatt, US 6,327,581 Bl filed 4/6/1998 discloses building a support vector machine 
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classifier. Gibbon et al, US 6,714,909 Bl filed 1 1/21/2000 discloses automated multimedia 
content indexing and retrieval. Bradshaw, US 2002/0122596 Al filed 1/2/2001 discloses a 
hierarchical, probabilistic, localized, semantic image classifier. Liu et al, US 2005/01 14325 Al 
continuation of application filed 10/30/2000 discloses semi-automatic annotation of multimedia 
objects. Luo et al., US 2002/0131641 Al provisional filed 1/24/2001 discloses determining 
image similarity. 

8. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1. 136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 571-272-4101 . The 
examiner can normally be reached on Mondays-Fridays 7:00am-3:30pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Heather R Herndon can be reached on 571-272-4136. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



PJS 

6/28/2005 




